This repository was archived by the owner on Jul 19, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 163
SAXParser concurrency bug causes list results to appear empty #546
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DETAILS: There is a concurrency bug in the XML parsing logic which can cause the results of a List Blobs
operation to appear empty.
Please refer to Utility.java#L131. Utility.saxParserThreadLocal is a ThreadLocal<SAXParser> with a SAXParserFactory
member variable. The factory is initialized in ThreadLocal.initialValue which then continues to use the
member variable reference. Because initialValue is called once for each thread, it is possible for
Thread A to be after the call to "factory.setNamespaceAware(true)" but before the call to "return factory.newSAXParser()"
when it yields to Thread B which calls "factory = SAXParserFactory.newInstance()" and then yields to Thread A which
then calls "return factory.newSAXParser()". Since this is a reference to the factory member variable, this instance
would therefore not be namespace aware since setNamespaceAware(true) has not been called. It therefore returns a
SAXParser that is not namespace aware.
A SAXParser that is not namespace aware will always return a list count of 0. Please refer to BlobListHandler.java#L84.
BlobListHandler.startElement takes the localName parameter and compares it with "Blob" on line 84, to find each blob item
in the list. When a SAXParser is not namespace aware, the localName parameter is always empty. Please refer to the
documentation for DefaultHandler.startElement, which describes the localName parameter as follows, "The local name
(without prefix), or the empty string if Namespace processing is not being performed." This was also confirmed by testing,
so the documentation is correct. Thus when the SAXParser is not initialized correctly (is not namespace aware) the results
of List Blobs operation will always appear to be empty.
TESTS: The test testSAXParserConcurrency was added to validate that the SAXParser returned by Utility.getSAXParser
is correctly configured, even when called under a highly concurrent load.
This test will occassionaly fail without the fix, but the probability of faiulre is very low. To increease the likelihood
of failure you can add calls to sleep before and after the call to "factory.setNamespaceAware(true)" in Utitlity.java as
shown below:
private static void sleep(long l) {
try {
Thread.sleep(l);
}
catch (InterruptedException e) {
}
}
/**
* Thread local for SAXParser.
*/
private static final ThreadLocal<SAXParser> saxParserThreadLocal = new ThreadLocal<SAXParser>() {
SAXParserFactory factory;
@OverRide public SAXParser initialValue() {
factory = SAXParserFactory.newInstance();
sleep(100);
factory.setNamespaceAware(true);
sleep(10);
try {
return factory.newSAXParser();
} catch (SAXException e) {
throw new RuntimeException("Unable to create SAXParser", e);
} catch (ParserConfigurationException e) {
throw new RuntimeException("Check parser configuration", e);
}
}
};
rickle-msft
approved these changes
Jun 22, 2020
ThomasMarquardt
added a commit
to ThomasMarquardt/hadoop
that referenced
this pull request
Jun 25, 2020
Contributed by Thomas Marquardt DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK. TESTS: A new test was added to validate the fix. All tests are passing: wasb: mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify Tests run: 248, Failures: 0, Errors: 0, Skipped: 11 Tests run: 651, Failures: 0, Errors: 0, Skipped: 65 abfs: mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 64, Failures: 0, Errors: 0, Skipped: 0 Tests run: 437, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
asfgit
pushed a commit
to apache/hadoop
that referenced
this pull request
Jun 25, 2020
Contributed by Thomas Marquardt DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK. TESTS: A new test was added to validate the fix. All tests are passing: wasb: mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify Tests run: 248, Failures: 0, Errors: 0, Skipped: 11 Tests run: 651, Failures: 0, Errors: 0, Skipped: 65 abfs: mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 64, Failures: 0, Errors: 0, Skipped: 0 Tests run: 437, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
asfgit
pushed a commit
to apache/hadoop
that referenced
this pull request
Jun 26, 2020
Contributed by Thomas Marquardt DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK. TESTS: A new test was added to validate the fix. All tests are passing: $mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify Tests run: 231, Failures: 0, Errors: 0, Skipped: 4 Tests run: 588, Failures: 0, Errors: 0, Skipped: 12 Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 37, Failures: 0, Errors: 0, Skipped: 0 Tests run: 407, Failures: 0, Errors: 0, Skipped: 34 Tests run: 151, Failures: 0, Errors: 0, Skipped: 19 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DETAILS: There is a concurrency bug in the XML parsing logic which can cause the results of a List Blobs operation to appear empty.
Please refer to Utility.java#L131. Utility.saxParserThreadLocal is a ThreadLocal with a SAXParserFactory member variable. The factory is initialized in ThreadLocal.initialValue which then continues to use the member variable reference. Because initialValue is called once for each thread, it is possible for Thread A to be after the call to "factory.setNamespaceAware(true)" but before the call to "return factory.newSAXParser()" when it yields to Thread B which calls "factory = SAXParserFactory.newInstance()" and then yields to Thread A which then calls "return factory.newSAXParser()". Since this is a reference to the factory member variable, this instance
would therefore not be namespace aware since setNamespaceAware(true) has not been called. It therefore returns a SAXParser that is not namespace aware.
A SAXParser that is not namespace aware will always return a list count of 0. Please refer to BlobListHandler.java#L84. BlobListHandler.startElement takes the localName parameter and compares it with "Blob" on line 84, to find each blob item in the list. When a SAXParser is not namespace aware, the localName parameter is always empty. Please refer to the documentation for DefaultHandler.startElement, which describes the localName parameter as follows, "The local name
(without prefix), or the empty string if Namespace processing is not being performed." This was also confirmed by testing, so the documentation is correct. Thus when the SAXParser is not initialized correctly (is not namespace aware) the results of List Blobs operation will always appear to be empty.
TESTS: The test testSAXParserConcurrency was added to validate that the SAXParser returned by Utility.getSAXParser is correctly configured, even when called under a highly concurrent load.
This test will occassionaly fail without the fix, but the probability of faiulre is very low. To increease the likelihood of failure you can add calls to sleep before and after the call to "factory.setNamespaceAware(true)" in Utitlity.java as shown below: